This is a good thing to do. Though, is a separate input flag needed? The code can check if warpx.eb_implicit_function has been specified, or eb2.geom_type. Elsewhere, enable type input parameters have been removed, relying on specification of the controlling parameters.

Thanks, Dave!

I was wondering as well, but in the end we have at least three ways to turn it on:
https://warpx.readthedocs.io/en/latest/usage/parameters.html#embedded-boundary-conditions

• warpx.eb_implicit_function
• warpx.eb_potential(x,y,z,t)
• and, currently undocumented, a way to load via STL file in AMReX

We could a) fix the lack of docs and b) write a helper function that checks them or so? Open to suggestions cc @RemiLehe @roelof-groenewald

The warpx.eb_potential(x,y,z,t) parameter would not be good to use since it is perfectly fine to leave that out - it is only used by the electrostatic solver and by default the embedded boundary potential is just 0.
I think it would be reasonable to then check if either warpx.eb_implicit_function or the STL file input is specified. We don't expect to have a lot more ways of loading EBs in the future, right? So it shouldn't be hard to maintain a list of or statements to determine the runtime parameter.

A separate bool ParmParse parameter is not needed unless we want to the one flag to rule them all. If we want a bool variable to be used in the code for convenience, we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

There are basically two ways to enable EB in WarpXInitEB, (1) warpx.eb_implicit_function (2) AMReX's eb2.geom_type. The STL way is the second way documented here https://amrex-codes.github.io/amrex/docs_html/EB.html?highlight=stl.

Thank you for all the ideas!

we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

In ImpactX and also more in WarpX, I try to avoid using global variables in the central sim class and instead use globals from the ParmParser. That way, we do not need to keep them in sync when changing options at runtime (e.g., from Python) and do not need to include a heavy central class everywhere.
But I also just introduced WarpX::m_eb_enabled... 😅

I would for now write a free standing helper function instead that we can use :)

Maybe a free function in WarpX like this

bool hasEB ()
{
#if defined(AMREX_USE_EB)
    return ! amrex::EB2::IndexSpace::empty();
#else
    return false;
#endif
}

Note that amrex::IndexSpace has a static member, a vector of amrex::IndexSpace. Somehow this comes into my mind by your mention of global variable. https://en.wiktionary.org/wiki/%E6%88%91%E4%B8%8D%E5%85%A5%E5%9C%B0%E7%8D%84%EF%BC%8C%E8%AA%B0%E5%85%A5%E5%9C%B0%E7%8D%84

Note that amrex::EB2::IndexSpace::empty() is true before WarpX::InitEB is called and will be false if EB is initialized in that function.

Awesome, that resolves the differences 🎉
No more failing azure integration tests ✨

Correctness is now ensured (see above & passing CI).

For performance, I benchmarked 3D runs on my laptop and an exclusive Perlmutter CPU node.
Somehow both machines were very noisy today, likely due to random overheads from MPI startups (srun) in the PM case and due to insufficient pinning on my laptop (and other background desktop noise).

From what I can see under the provided noise, there is now no significant performance drawback beyond the noise level for runs that compile with EB but do not use it.

Data & Notebook

2024-09-06_EB-runtime.zip

Perlmutter CPU Node

Compiled and ran in local in-memory file system /dev/shm/.
Allocated via: salloc --exclusive --nodes 1 --qos interactive --time 02:00:00 --constraint cpu --account=m4272 --ntasks-per-node=2
Compiled via: cmake --fresh -S . -B /dev/shm/build_cpu_rteb -DWarpX_COMPUTE=OMP -DWarpX_FFT=ON -DWarpX_DIMS="3" -DMPIEXEC_PREFLAGS="--cpu-bind=cores"
Ran via: ctest --test-dir /dev/shm/build_cpu_rteb -E AMReX -R run

Laptop

Memory footprint evaluation

I ran Examples/Physics_applications/laser_acceleration/inputs_base_3d, which does not use EBs at runtime, both on development (using WarpX_EB=OFF) and on this PR (using the new default, WarpX_EB=ON).
I used 1 MPI process and one OMP process to avoid fluctuations.

Development Memory Footprint

Pinned Memory Usage:
-------------------------------------------------------------------------
Name                                    Nalloc  Nfree    AvgMem    MaxMem
-------------------------------------------------------------------------
The_Pinned_Arena::Initialize()               1      1  1948   B  8192 KiB
ParticleContainer::addParticles           1984   1984  2411   B  5144 KiB
WarpX::InitData()                           52     52     8   B   131 KiB
main()                                      75     75   799   B   800   B
PhysicalParticleContainer::AddPlasma()       2      2   159   B   160   B
-------------------------------------------------------------------------

Cpu Memory Usage:
---------------------------------------------------------------------------
Name                                      Nalloc  Nfree    AvgMem    MaxMem
---------------------------------------------------------------------------
WarpX::InitData()                             40     40    54 MiB    54 MiB
Diagnostics::FilterComputePackFlush()         50     50   295 KiB  9448 KiB
ParticleContainer::BuildRedistributeMask      16     16  5395 KiB  7892 KiB
FabArray<FAB>::SumBoundary_nowait()         1200   1200   249 KiB  4275 KiB
WarpX::SyncCurrent()                        1200   1200   665 KiB  4275 KiB
WarpX::shiftMF()                            2364   2364  1274 KiB  4275 KiB
LaserParticleContainer::Evolve()               3      3   260 KiB   264 KiB
PhysicalParticleContainer::Evolve()            3      3   260 KiB   264 KiB
Filter::ApplyStencil(MultiFab)               709    709    12 KiB   106 KiB
---------------------------------------------------------------------------

PR Memory Footprint

Pinned Memory Usage:
-------------------------------------------------------------------------
Name                                    Nalloc  Nfree    AvgMem    MaxMem
-------------------------------------------------------------------------
The_Pinned_Arena::Initialize()               1      1  1542   B  8192 KiB
ParticleContainer::addParticles           1984   1984  2337   B  5144 KiB
WarpX::InitData()                           52     52    11   B   131 KiB
main()                                      75     75   799   B   800   B
PhysicalParticleContainer::AddPlasma()       2      2   159   B   160   B
-------------------------------------------------------------------------

Cpu Memory Usage:
---------------------------------------------------------------------------
Name                                      Nalloc  Nfree    AvgMem    MaxMem
---------------------------------------------------------------------------
WarpX::InitData()                             40     40    54 MiB    54 MiB
Diagnostics::FilterComputePackFlush()         50     50   292 KiB  9448 KiB
ParticleContainer::BuildRedistributeMask      16     16  5392 KiB  7892 KiB
FabArray<FAB>::SumBoundary_nowait()         1200   1200   241 KiB  4275 KiB
WarpX::SyncCurrent()                        1200   1200   653 KiB  4275 KiB
WarpX::shiftMF()                            2364   2364  1253 KiB  4275 KiB
LaserParticleContainer::Evolve()               3      3   260 KiB   264 KiB
PhysicalParticleContainer::Evolve()            3      3   260 KiB   264 KiB
Filter::ApplyStencil(MultiFab)               709    709    12 KiB   106 KiB
---------------------------------------------------------------------------

Summary

The maximums do not change, this looks good to me and we seem to effectively prevent allocating EB-related fields if we do not use them.